System and methods for determining health-related metrics from data collected by a mobile device

ABSTRACT

Techniques and systems include predicting various health conditions using a photoplethysmography (PPG) signal or a video signal based on images of a patient&#39;s fingertip or other tissue or other body portion captured using a mobile device, such as a smartphone or tablet. The video signal may be transformed into a pseudo PPG signal to measure blood volume changes in the patient&#39;s blood flow to derive data indicating a disease state or health-related characteristic, such as blood oxygen level, blood glucose level, heart rate variability, hemoglobin, respiration rate, or arrhythmia. Techniques involve real-time environment assessment and problematic issue detection, training an artificial intelligence (AI) model to measure signal quality so as to select high-quality signals from a range of signals, and domain adaption and transfer learning to make use of publicly available datasets.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority and benefit from the U.S. ProvisionalPatent Application 63/226,541, filed Jul. 28, 2021, and titled, “Systemand Methods for Determining Health-Related Metrics from Data Collectedby a Mobile Device,” which are incorporated herein by reference in theirentirety for all purposes.

BACKGROUND

There are several quantities that are regularly measured and evaluatedto provide information about a person's health. These quantities andboth their real-time values and trends over various timescales cansometimes be used as indications of disease or a serious healthcondition. Examples of such quantities that may be used to monitor and,in some cases, diagnose a health condition include blood pressure, pulseor heart rate, and blood oxygen saturation (SpO2), among otherbiomarkers. Conventionally, such health-related quantities are measuredat a health clinic, the office of a provider of medical services, orwith special equipment that is used in the home.

However, these conventional methods of measuring health quantities arenot always available, may require advance planning to use, and are notconvenient for situations where a person is engaging in activities awayfrom home. Further, the conventional methods provide a “snapshot” intime of the values of the health-related quantities but may not providesufficient data or data collected at a sufficient time or set of timesto detect or diagnose a health condition. For example, a physician orparamedic may want to know a person's blood pressure over a timeinterval around when they suffered some type of pain. This may bedifficult to do without the person being in a location where someone canmake those measurements. As another example, a person may want tomonitor how their blood pressure responds to certain activities orbehaviors, and such monitoring may be difficult to do outside of theirhome.

Although there are some portable devices that can be used to monitorblood pressure, these typically are not as accurate as desired. Further,although portable blood pressure monitors are available, these typicallyrequire that a person be in a situation where a “cuff” may be used toobtain the blood pressure reading, and this is not always possible orconvenient.

What is desired are systems, apparatuses, and methods for enabling aperson to monitor one or more of their health-related quantities moreconveniently using a ubiquitous mobile device and without having to beat a particular location.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure will be understood more fully from the detaileddescription given below and from the accompanying figures of embodimentsof the disclosure. The figures are used to provide knowledge andunderstanding of embodiments of the disclosure and do not limit thescope of the disclosure to these specific embodiments.

FIG. 1 illustrates a user engaging a camera of a smartphone formeasuring a biomarker or other health-related quantity, according tosome embodiments.

FIG. 2 illustrates example photoplethysmography (PPG) signal waveformsand a comparison between a PPG signal obtained from a pulse oximeter anda signal derived from the red channel of an RGB video signal, accordingto some embodiments.

FIG. 3 is a block diagram illustrating a process flow and functionalelements that may be used in implementing various embodiments.

FIG. 4 illustrates a process that uses a baseline model incorporating anaveraged red video channel to create a time-series signal that resemblesa PPG signal, according to some embodiments.

FIG. 5 illustrates a process that uses a baseline model incorporating atime-series of video frames that are passed directly to a deep learningmodel to predict a biomarker, according to some embodiments.

FIG. 6 is a flow diagram of a process for addressing signal quality of aPPG signal, according to some embodiments.

FIG. 7 is a flow diagram of a process for determining signal quality ofa PPG signal and determining a disease state or biomarker based on thePPG signal, according to some embodiments.

FIG. 8 illustrates elements or components of a computer device or systemconfigured to implement a method, process, function, or operation ofsome embodiments described herein.

FIG. 9 illustrates a Software-as-a-Service (SaaS) system in which someembodiments described herein may be implemented.

FIG. 10 illustrates elements or components of an example operatingenvironment in which some embodiments described herein may beimplemented.

FIG. 11 illustrates some details of elements or components of amulti-tenant distributed computing service platform, of which someembodiments described herein may be implemented.

DETAILED DESCRIPTION

Photoplethysmography (PPG) is an optical technique used to detectvolumetric changes in blood in peripheral circulation, usinglow-intensity infrared (IR) light. When light travels through biologicaltissues it is absorbed by bones, skin pigments, and both venous andarterial blood. Since light is more strongly absorbed by blood thansurrounding tissues, changes in blood flow can be detected by PPGsensors as indicated by changes in the transmitted intensity of light. Avoltage signal from PPG sensors may be proportional to the quantity ofblood flowing through blood vessels. Even small changes in blood volumecan be detected using this method. A PPG signal has several componentsincluding volumetric changes in arterial blood that is associated withcardiac activity, variations in venous blood volume that modulates thePPG signal, and a DC component showing the tissues' optical propertiesand subtle energy changes in the body.

This disclosure describes a number of techniques and systems fordetermining or predicting various health conditions. The techniques andsystems involve use of an optical signal, such as a PPG signal or avideo signal transformed into a pseudo PPG signal, to measure bloodvolume changes in a patient's blood flow to derive data indicating adisease state or biomarker. In the past, an optical signal used for thispurpose generally suffered from quality and inconsistency issues, evenin a controlled environment on the same patient.

Previous efforts to derive blood pressure from smartphone-sourced PPGhave shown promising results in lab settings but have not reached goodenough accuracy in real-world settings for widespread adoption. Asubstantial technical barrier to achieving sufficient real-worldaccuracy is the handling of the inconsistency and quality of these PPGsignals created under a diversity of conditions set forth by sensors(e.g., camera hardware, numerous flash settings), users (e.g., bodyweight, skin tone, ages), and environments (e.g., contact grips,lighting conditions). There is no canonical “ground truth” PPG, sinceeven purpose-built PPG devices in controlled environments may producedifferent signals for the same subject by small changes in placement ororientation of the device.

Machine learning (ML) has been used to attempt to address the issues andcomplexity described above. Traditional ML methods such as a SupportVector Machine (SVM) algorithm or regression have shown promisingresults but require perhaps more than fifty or so features extracted byhand, even when using PPG signals from purpose-built devices incontrolled environments. In recent years, neural networks have been usedin different ways to successfully improve performance, especially withsmartphone-sourced PPG. But what has been achieved in this way is stillinsufficient due to the large diversity of signals and expensive,limited training data. One common theme is that poor signals are oftenthrown out during training and are not considered during inference,leading to many failed or inaccurate readings. Another common theme isthat models may work only on a limited number of smartphones or otherdevices.

Embodiments described herein address the problems discussed above andmay achieve real-world accuracy with even just limited data. Theembodiments involve real-time environment assessment and problematicissue detection, training an AI model to measure signal quality so as toselect high-quality signals among signals having a range of qualities,and domain adaption and transfer learning to make use of publiclyavailable datasets, just to name a few examples.

In particular, embodiments described herein are directed to systems,methods, and apparatuses for more conveniently and accurately measuringand monitoring one or more health-related characteristics, including butnot limited to blood pressure and heart rate, for example. In someembodiments, the health-related characteristics may include blood oxygenlevel, blood glucose level, hemoglobin A1C, heart rate variability,hemoglobin, respiration rate, blood oxygen, or arrhythmia.

One embodiment is directed to a method for determining one or morehealth-related quantities for a person based on video images capturedusing a mobile device (e.g., a smartphone). The method may includesteps, stages, or operations that include activation or launch of anapplication hosted by the mobile device and collecting a sample orsamples of video of a person's fingertip or tissue from another part ofthe body. For example, during data collection, participants may haveequipment placed on them, such as a pulse oximeter placed on the finger,toe, and/or earlobe, a single-lead Electrocardiogram (ECG), or anInertial Measurement Unit (IMU) taped to the sternum. Collection of datafrom these sensors may include a participant's blood pressure, which maybe taken using the auscultation method (e.g., using a stethoscope tolisten for Korotkoff sounds).

The method may also include processing the collected sample(s) toapproximate a PPG signal and detection of possible data collectionproblems during acquisition of user data, which can occur under normaluse in diverse conditions. The method may further include evaluatingquality of the PPG signal that may have been gathered from diversesubjects and cameras under diverse conditions and passing the PPG signalas input to a trained blood pressure AI model.

The activation or launch of the application may involve the applicationsetting collection parameters and equipment characteristics, such asdevice manufacturer and model, camera or video settings, lens, or videoprocessing options, and so on. If needed, in some embodiments, theapplication may collect data regarding the patient, such as skin tone,and other data regarding local environment, lighting conditions,stability/motion of device, and so on, before beginning video capture.Collecting a sample or samples of video from a person's fingertip (orother tissue) using a mobile device may involve use of the application'suser interface (UI) to guide a user in the placement of their fingertipand in holding the mobile device correctly and steadily to collect thedata. A camera (and flash LED) of a mobile device, such as a smartphone,or other portable device may be used.

Processing the collected sample(s) to approximate a PPG signal mayinvolve a PPG signal generator module or similar functionality that maycontinuously process raw camera frames by extracting average red (R),green (G), and blue (B) values of all the RGB pixels in each videoframe. In some implementations, the red channel is taken to approximatethe PPG signal. Obtained red channel signals may be smoothed using aButterworth filter or other filter and up-sampling may be performedusing linear interpolation, for example. A Butterworth filter is a typeof signal processing filter designed to have a frequency response asflat as possible in the passband. It may also be referred to as amaximally flat magnitude filter.

In some implementations, data collection problems (problematic issues)during acquisition of user data may be detected. Such problems may occurunder normal use in diverse conditions. In other words, issues orfactors that could result in unreliable or unusable data may bedetected. For example, ambient conditions or factors may include motionof camera (e.g., relative to finger), movement of finger (e.g., relativeto camera), improper position of finger, and a change in lighting thatimpacts an ability to accurately and reliably extract a PPG-like signalfrom camera frames. An Issue Detector module or such functionality maybe included in the system architecture, as described below.

Camera settings may be adjusted to get a “good” (e.g., acceptable)signal during diverse conditions (ambient lighting, skin tone of user,mobile device camera, etc.). In some implementations, previouslymeasured or entered skin tone information of a user may be retrievedfrom memory of the user's mobile device. An output that includes adetermined disease state or a biomarker may be modified based on theretrieved skin tone information. In some embodiments, to at leastpartially control ambient factors/conditions, an Environment Assessormodule may step up the light sensitivity periodically (e.g., every 1.2seconds), subsequently holding constant the light sensitivity when thered channel intensity, for example, from the camera frames reach aminimum (predetermined) threshold. The Environment Assessor module mayalso adjust the intensity of light emitted by a flash LED system of thecamera.

Evaluating the quality of the PPG signal gathered from diverse subjectsand cameras under diverse conditions may involve identifying individualpulses in the PPG signal. For example, in one embodiment, a PulseGeneration module or such functionality may identify a peak or valleythat expresses an individual pulse in the PPG signal. A signal qualityevaluation process may reject or remove from further processing pulsesthat fail to meet the threshold required for processing and evaluation.In one embodiment, a Signal Quality Assessor module may be a neuralnetwork trained to assess individual pulses. During training of theSignal Quality Assessor module, collected fingertip data (or similartype of data for other tissue or body part) may be manually annotated(described below) with an “acceptable” or “not acceptable” label toproduce training data for a model. For example, an “acceptable” pulse isuniform (multiple pulses are consistent among themselves) and has thecharacteristics of a PPG signal (e.g., generally having a steep slope upto a relatively sharp peak and having zero or up to 2 smaller notches onthe way down).

Based on environmental and signal quality conditions (during signalacquisition), and an absence of data acquisition problems, the PPGpulses may be passed as input to a trained AI blood pressure model. Inone embodiment, the blood pressure model is a neural network thatcontinuously takes in “acceptable”-quality pulses from the PPG signal asinput and produces systolic blood pressure (BP) and diastolic BP asoutput. The model averages the produced systolic and diastolic BP valuesto produce final BP values that are returned to the user.

As mentioned above, techniques that may achieve real-world accuracy mayinvolve training an AI model to measure signal quality so as to selecthigh-quality signals from signals having a range of qualities. In oneembodiment, an AI model training process may involve, due to limitedtraining data, a base model that is trained on publicly available,synchronized, PPG and BP datasets. The trained base model may be used asa basis for continued training using learning on collected (e.g.,historical) pulse oximeter PPG, smartphone fingertip video, and bloodpressure data, as described below. Although examples described hereininvolve a fingertip, other tissue or parts of the body may be usedinstead of, or in addition to, the fingertip. Claimed subject matter isnot limited in this respect. A smartphone-sourced PPG signal producedfrom fingertip videos may also be used by applying a Butterworth filter,and up-sampling the pulse oximeter PPG and smartphone-sourced PPG tomatch the frequency of the publicly available data using linearinterpolation, for example.

Additional AI training data may be captured for use in training the basemodel, such as from users of the application, training data collectedunder controlled conditions, and continuous data acquisition and modeltraining through data collection along with accurate measurement ofhealth quantity using a separate device(s). When in use, data collectionand processing may be performed on a user's device, wherein the user maymaintain privacy.

Output from the trained AI model may provide information representing ameasure of one or more of the person's blood pressure, heart rate, SpO2,or other health-related quantity. Based on the model output, one or moretasks may be executed by an application hosted by a mobile device, suchas providing results via an application programming interface (API) callback that enables a developer to use the data in their own application,showing trends in the measured quantities over time, tuning medicationdosages based on changes in the measured quantities, placing themeasured data into an electronic health record (EHR) for use by aphysician, and statistically analyzing the measured data in conjunctionwith other data or information (such as sleep quality, stress, oranxiety levels, etc.). Other tasks that may be performed by a mobiledevice, for example, may include generating an alert to a user that ameasured quantity exceeds a preset threshold value, wherein the averagevalue of a measured quantity exceeds the preset value over a set timeperiod, and wherein the measured value in combination with anotheraspect of the user (BMI, weight, blood glucose level, etc.) suggests aproblem needing attention. Still other tasks may include generating aphone call to an emergency number if a measured quantity alone or overtime exhibits a characteristic of serious medical problem (such asstroke, etc.) and generating a recommendation to engage in an activity,undertake a change in behavior, etc.

As described above, in various embodiments, biomarkers may be estimatedfrom biosensor data, which may be an optical signal like a PPG,electrical signals like an ECG, and so on. The optical biosensor datamay be acquired from a smartphone-captured video of a person's finger orfrom a PPG sensor in devices such as a pulse oximeter. For example,measurements of real-time blood pressure and heart rate data may beacquired via fingertip video obtained with a mobile device. In animplementation, the mobile device may include a data measurement module(DMM), which may comprise a software development kit (SDK) embedded intoan operating system of the mobile device. A user may use the mobiledevice (e.g., a smartphone, wrist-worn wearable such as smartwatch orsmart wristband, earbud, or tablet) to measure their blood pressure andheart rate, for example, by operating the DMM, which may navigate theuser to a particular view (e.g., menu of options and/or instructions) onthe smartphone. The user may place their fingertip against the lens ofthe mobile device camera, and after about a minute or so, for instance,the DMM may provide the user's blood pressure and heart rate data to anapplication on the mobile device or external to the mobile device. Insome implementations, video frames and related data may be processedcompletely on the device so that user identities may be completelyunknown outside the device, for anonymity.

In some embodiments, the DMM works by extracting the red channel fromeach frame of the fingertip video to form a PPG signal. As explainedabove, PPG is commonly used in methods for measuring blood volumechanges and can be used to derive health measures like heart rate, Sp02,and blood pressure. A neural network may be used to extract these healthmeasures from the smartphone signal produced by the DMM.

FIG. 1 illustrates a data gathering process 102 involving tissue of afingertip 104 of a user engaging a camera 106 of a smartphone 108 formeasuring a biomarker or other health-related quantity, or fordetermination of a health condition, according to some embodiments. Inother example embodiments, other tissue or portion of a body may beused. Claimed subject matter is not limited to measurements based on afingertip. In the situation illustrated, a camera lens on the back ofsmartphone 108 is used. Generally, fingertip 104 may be heldsubstantially steady on or over camera 106 (e.g., or its lens) forseveral seconds to a minute or so while a biomarker application, forexample, hosted by smartphone 108 causes the camera to capture images ofthe fingertip. Images may be captured at a frequency of milliseconds orseconds, depending on instructions from the biomarker application. Forexample, a frequency of image capture may be up to several dozen framesper second if the images are to be analyzed or processed as a videosignal.

Conditions that exist during image capture may be important and thusconsidered during image or video processing, as explained below.Lighting conditions, motion conditions, and orientation of thesmartphone may be important by affecting the quality of the images orthe video that will subsequently be used to arrive at a pseudo PPGsignal. Smartphone 108 may include an onboard flash LED 110, an onboardlight sensor 112, a capacitive touch sensor 113, and an onboardaccelerometer 114 (or IMU). Flash LED 110 may provide light to thefingertip and the surrounding area. This may light illuminate fingertip104 with an intensity that allows for successful image capture by camera106. The biomarker application hosted by the smartphone may at leastpartially control the intensity of light emitted by flash LED 110. Lightsensor 112 may measure the intensity of ambient light 116 impinging onthe light sensor. This intensity may be indicative of illumination offingertip 104 and the surrounding area. Light sensor 112 may provide itsmeasurements to the biomarker application hosted by the smartphone. Inturn, the application may control the flash or exposure settings of thecamera based on the light sensor measurements.

Capacitive touch sensor 113 may produce a signal based on whether afingertip or other tissue is in contact with the camera (or a lens orlens cover thereof) and accelerometer 114 may measure motion andorientation of smartphone 108. Such motion may be vibratory motion orlonger time-constant motion, for example. For successful image capturethat can lead to high-quality pseudo PPG signals, relative motionbetween fingertip 104 and camera 106 should be small or non-existent. Asdescribed below, measurements by light sensor 112, capacitive touchsensor, and accelerometer 114 may be used by the biomarker applicationto assess the quality of captured images, video, or resulting pseudo PPGwaveforms. For example, such measurements may be grouped as metadata forthe captured images or video. Accordingly, the metadata can be used todetermine the conditions that existed during acquisition of the imagesor video.

FIG. 2 illustrates example PPG waveforms 202 and illustrates thesimilarity between a PPG signal 204 obtained from a pulse oximeter and ared channel signal 206 derived from the red channel of a video signalcaptured by a smartphone (e.g., 108) or other similar portable device.As can been seen from the figure, the red channel signal 206, whichcorresponds with blood flow in the fingertip, is close to and mimics thebehavior of PPG signal 204 when environment and signal quality areadequately controlled. This allows for the red channel signal being asuitable substitute for an actual PPG signal and provides a basis forwhich signal capture and signal quality techniques described herein areutilized.

FIG. 3 is a block diagram of a system 300, illustrating a process flowand components that may be used in implementing an embodiment of thedisclosed methods. For example, system 300 may be a DMM that includessub-systems, elements, components, or modules that implementfunctionality for measuring blood pressure, among other biomarkers.Specifically, system 300 may include a client application 302, aBiomarker Detector user interface (Biomarker Detector UI) 304, a PPGSignal Generator 306, an Issue Detector 308, and a Biomarker Detectionmodule 310. Client application 302 may be similar to or the same as thebiomarker application described above.

Client application 302 may comprise computer-executable instructionsthat are installed on a user's device, which may be a mobile device suchas a smartphone, wrist-worn wearable such as smartwatch or smartwristband, earbud, or tablet. In some embodiments, the application usesAPIs to integrate functionality for measuring blood pressure or otherhealth-related quantities with the capabilities of the mobile device.Client application 302 may use the APIs to navigate to a user interfaceassociated with a video/data collection service and to provide bloodpressure values when complete. In some embodiments, client application302 may be developed using an SDK that is provided to enable access to,and use of, models and data processing methods, described below.

Biomarker Detector UI 304 is a user interface component that guides(e.g., provides real-time instructions for) the user in measuring theirblood pressure or other biomarkers. Biomarker Detector UI 304 (andunderlying services or functions) may provide tools to enable the userto initiate the video/data collection used to determine their bloodpressure and/or provide a mechanism to interact with other features orfunctionality of system 300 and methods described herein. For example,the Biomarker Detector UI may display a view that provides instructionsand shows feedback to the user, captures camera frames, and continuouslyor periodically passes the frames to PPG Signal Generator 306. Forexample, Biomarker Detector UI 304 may guide the user in placing theirfingertip correctly on the camera, holding the smartphone correctly andsteadily, and may provide the user with measurement progress and mayindicate any errors in the process. Client application 302 may determinesuch measurement progress based, at least in part, on measurementsreceived from accelerometer 114, for example. Biomarker Detector UI 304,via client application 302, may keep the camera flash LED on, and maycontrol its intensity, to illuminate the finger during image capture.

Biomarker Detector UI 304 may be part of a service or application thatinteracts with Issue Detector 308 to control the acquisition ofvideo/data and processing of the acquired video/data into a PPG signalby PPG Signal Generator 306. In some embodiments, Biomarker Detector UI304 may be provided as a part of an SDK. In some embodiments, BiomarkerDetector UI 304 may allow for measurements of a different health-relatedquantity (e.g., other than, or in addition to, blood pressure). PPGSignal Generator 306 may provide PPG signals to Issue Detector 308 andBiomarker Detection module 310.

A blood pressure detection process, performed by Biomarker Detectionmodule 310, may begin once no issues (e.g., as determined by IssueDetector 308) have been identified during an initial time period, suchas the first five seconds of the process, for example. A PPG signal maybe passed through the following components to produce blood pressure: anEnvironment Assessor 312, a Signal Quality Assessor 314, and a Biomarkermodel 316. There generally may be many variables that impact the qualityof the PPG signal that in turn impact the accuracy of Biomarkerreadings. Such variables may include ambient lighting conditions, skintone, and smartphone camera quality. To control for this, EnvironmentAssessor 312, via client application 302, may hold constant the lightsensitivity and shutter speed of the camera (e.g., 106) of the mobiledevice at values that produce a minimum threshold of intensity. In someimplementations, Environment Assessor 312 may step up the lightsensitivity periodically, such as every 10 seconds, for example. Inanother implementation, Environment Assessor 312 may step up the lightsensitivity by 75 ISO every 1.2 seconds, locking the light sensitivitywhen the red channel intensity from the camera frames reaches a minimumthreshold. These values (e.g., 75 ISO and 1.2 s) may be determinedempirically to increase the red channel intensity as fast as possiblewhile stabilizing at each increment. Claimed subject matter is notlimited to such values.

An updated PPG signal with camera settings (e.g., as metadata, such astime stamp of image capture, exposure time, aperture, resolution, and soon) are passed through Signal Quality Assessor 314, which may produceindividual pulses from signals passed from Environment Assessor 312 andmay pass acceptable pulses to Biomarker model 316. In one embodiment,Signal Quality Assessor 314 is a neural network trained to assessindividual pulses. Training data may be manually annotated based oncollected user fingertip data with a label of “acceptable” or “notacceptable” to produce training data for this model. In general, an“acceptable” pulse is uniform (consistent with other pulses) and has thecharacteristics of a PPG signal (generally a steep slope up to arelatively sharp peak and has 0 to 3 smaller notches on the way down),though claimed subject matter is not so limited.

Biomarker model 316, which is a neural network, may depend on relativelyhigh quality signals to produce a good result. In some implementations,a time-series of image frames of video are partitioned into video chunks(e.g., segments) that each have a predetermined time span. Accordingly,Signal Quality Assessor 314 looks at such chunks of signals provided byEnvironment Assessor 312 and only passes acceptable signals to Biomarkermodel 316. For example, such chunks of signals may be two-second longsegments of a PPG signal. In some implementations, two contiguous videochunks may be combined into a video segment having a portion thatcomprises an overlap between the two contiguous video chunks. SignalQuality Assessor 314 may generate a quality score determined bycomparing a signal quality feature of a PPG signal provided to BiomarkerDetection 310 to a same-type signal quality feature of a deeplearning-based model PPG signal, which may be stored in computer memoryor updated and provided by Biomarker Model 316, for example. In otherimplementations, Signal Quality Assessor 314 may generate a qualityscore based on metadata included in the PPG signal provided to BiomarkerDetection 310.

In some implementations, Signal Quality Assessor 314 is a neural networktrained to assess chunks of PPG signals. Biomarker Model 316 may producea systolic BP and diastolic BP as output. The model may average theproduced systolic and diastolic BP values to produce final BP valuesthat are returned to the user. After acceptable PPG/red channel signalshave been processed, the blood pressure values may be provided to theuser through client application 302. In some embodiments, the bloodpressure values, peak values, changes over time, and so on may be usedas input data to a set of rules or a trained model to determine whetherto generate an alert, place a call to a medical provider, recommend anactivity or change in behavior to the user, etc. In some embodiments,Biomarker model 316 is based, at least in part, on a neural network thatis trained using transfer learning on pulse oximeter PPG data andinformation collected or produced by a mobile device hosting clientapplication 302, as described below. Such information may include videochunks or blood pressure data previously determined by the mobiledevice, for example.

In some embodiments, as mentioned above, fingertip data, such as in theform of PPG segments, may be annotated (automatically or manually) asbeing “acceptable” or “not acceptable” to produce training data forBiomarker model 316, which is a neural network that continuously orperiodically takes in as input, “acceptable”-quality, segments (e.g.,two-second long chunks) of the PPG signal and produces systolic BP anddiastolic BP as output. Biomarker model 316 may average the producedsystolic and diastolic BP values to produce final BP values that arereturned to the user. In some implementations, the training process forthe Biomarker model 316 may be performed by first training a base modelon publicly available, synchronized, PPG and BP datasets. The trainedbase model may continue to be trained using transfer learning on pulseoximeter PPG, smartphone fingertip video, and blood pressure datasubsequently collected. The smartphone PPG may be produced fromfingertip videos in a similar process on the mobile device. For example,this process may include using red channel data of a PPG, applying aButterworth filter, and up-sampling pulse oximeter PPG and smartphonePPG to match the frequency of the publicly available data using linearinterpolation.

PPG Signal Generator 306 may receive video frames collected by a camerain the user's device and may perform the signal/image processing togenerate the PPG/red channel signal for further analysis and evaluation.For example, PPG Signal Generator 306 may continuously or periodicallyprocess raw image frames by extracting the average red, green, and bluepixel values across an array of pixels in each frame. In someimplementations, a single image frame at a time is processed. In otherimplementations, two or more image frames are processed at a time. Thered values may be smoothed with a Butterworth filter and up-sampled vialinear interpolation for Biomarker model 316. Such processed frames andassociated PPG signal may be passed to the UI of client application 302as “detection complete,” as indicated by arrow 318.

Issue Detector 308 may be responsible for implementing a set ofprocesses to determine if the video/data collection should beinterrupted, stopped, or discarded due to unreliability caused by one ormore of the user's finger position, motion of the camera or user'sfinger, or ambient light conditions, among other possible sources oferror. Patient physiological data, in the form of a PPG signal, forexample, may include metadata, which may be measured by an IMU of thecamera, or other elements that may be used as signal quality features.The patient physiological data, which may also include a disease stateor a biomarker feature, may be provided to Issue Detector 308 from PPGSignal Generator 306, for example. Signal quality features and diseasestate or biomarker features may be manifested as characteristics orparameters of the PPG signal, for example. For example, a PPG signalhaving such characteristics and parameters may be provided to IssueDetector 308 to determine signal quality (e.g., a quality score) orwhether problematic issues occurred during the measurement process thatled to the PPG signal. The PPG signal having such characteristics andparameters may also be provided to Biomarker Detection 310 to determinea disease state or biomarker. For example, a process for determining thedisease state or biomarker of patient physiological data based on adisease state or a biomarker feature and a quality score may includedetermining that the quality score exceeds a threshold acceptance valueand determining the disease state or the biomarker of the patientphysiological data based on the disease state or biomarker feature.

Issue Detector 308 may use such metadata to determine ambient orenvironmental conditions that existed during the measurement processthat produced the patient physiological data. For example, metadatameasured by an IMU (e.g., or an accelerometer) of the camera may includerelative motion between the camera and a user's fingertip and theorientation of the camera (e.g., and thus the orientation of the mobiledevice housing the camera).

In some embodiments, translational and rotational motion of the cameramay be represented by one or more motion signatures, which may berepresented by a waveform or plot of values. For example, translationalmotion may be a plot or table of time-dependent values of displacementof the camera relative to a fixed point in space. The plot or table maylead to a motion signature for the camera during a time span. Similarly,rotational motion may be one or more plots or tables of time-dependentvalues of pitch, yaw, and roll of the camera relative to a fixed axis.The plots or tables may lead to other types of motion signatures for thecamera during a time span. Various features of the motion signatures maybe analyzed. Such features may include frequency-of-motion (e.g.,Fourier analysis of motion signatures in the frequency domain), shapesof waveforms or curves, and duration of such features (e.g., when theyoccurred relative to the time of data acquisition and for how long theyoccurred), just to name a few possible features. For example, a motionsignature expressing displacement of the camera may include a featurethat indicates that the camera was not resting on a stable surface(e.g., a tabletop). Such a feature may be a relatively low frequency(e.g., 5-10 Hz) component of the motion signature, which may be a resultof the camera being hand-held during data acquisition. This type ofmotion may likely adversely affect the quality of data acquired duringthe time of this motion because the proximity of the fingertip to thecamera may be difficult to hold constant with this type of motion. Inanother example, a motion signature expressing angular displacement,such as a relatively slow roll, of the camera may include a feature thatalso indicates that the camera was not resting on a stable surface. Thistype of motion, however, may not affect the quality of data acquiredduring the time of this motion because the proximity of the fingertip tothe camera may nevertheless be held constant with this type of motion.Such dynamics of camera motion, as given in these and other examples,may be captured in motion signatures. Accordingly, analyzing thesedynamics and how they affect motion of a fingertip or other tissueduring data acquisition may allow for assessing the quality of the data.

In other embodiments, relative motion between the fingertip and thecamera may be determined based, at least in part, on intensitymeasurements of pixel data of multiple image frames. For example,substantial changes of light intensity among image frames may indicateto Issue Detector 308 that relative motion occurred during capture ofthese image frames. In still other embodiments, relative motion betweenthe fingertip and the camera may be determined based, at least in part,on motion signatures in combination with intensity measurements of pixeldata of multiple image frames. In still other embodiments, relativemotion between the fingertip and the camera may be determined to be zeroif a signal from a capacitive touch sensor (e.g., 113), which may beonboard the handheld device and relatively near or on the lens of thecamera, indicates continuous contact between the fingertip and thecamera during data acquisition.

If an issue is detected, then Issue Detector 308 may send a signal ormessage to Biomarker Detector UI 304, as indicated by arrow 320, todiscard or prevent the further processing of particular samples of thecollected frames of video/data. Moreover, Issue Detector 308 may send asignal or message to Biomarker Detector UI 304 including instructionsfor the Biomarker Detector UI to adjust in real-time a parameter ofambient conditions, the adjusting being based on the determination, bythe Issue Detector, that the signal characteristic of the pseudo PPGsignal does not meet a quality criterion. The ambient conditions mayinclude relative motion between the camera and the user's finger and theambient light to which the sensor is exposed. Such adjustments inreal-time may occur during the measurement process.

In particular, Issue Detector 308 may continuously or periodically lookfor issues that prevent production of high-quality signals. System 300may include a detector, or process of detection, for each problematicissue, which, upon detection, may be provided as feedback to the user,via the UI of client application 302. For example, if a detector detectsa problematic issue, the user may be alerted and Biomarker Detectionmodule 310 may be reset.

As mentioned above, some examples of problematic issues may include afinger-off condition, motion/unstable ambient lighting condition, andphone orientation problem, just to name a few examples. In a particularimplementation, a finger-off condition may be detected by gathering thelast 500 milliseconds (ms) of video frames and analyzing the ratios ofred to blue and green to identify whether the user's finger is correctlyon the camera. A threshold for the ratio values may be dynamicallyselected based on the overall intensity in the collected frames toaccount for variables such as skin tone and camera settings. Thesethresholds may be identified empirically through collected fingertipvideo data where fingers are determined to be on or off at differentintensities and camera settings. The image frame may also be segmentedinto quadrants (e.g., or other proportions), with ratios and thresholdsbeing determined for each quadrant, with finger-off condition beingidentified if at least one quadrant's ratio is below a threshold. Inanother implementation, a finger-off condition may be detected byanalyzing motion signatures of the camera, as described above.

In a particular implementation, a motion/unstable ambient lightingcondition may be detected by gathering the last 500 ms of video framesand looking for the difference between minimum and maximum red pixelvalues. If the difference is more than a threshold value (e.g., 20 in arange of 0-255), it may be determined that too much motion has occurredor that the ambient lighting is too unstable. A phone orientationproblem may occur if the user does not hold the phone relatively flat,thus affecting the amount of ambient light that reaches the camera. Anon-board accelerator in the phone may be used to determine if the phoneis flat, for example.

Assuming that no problematic issues, such as those described above, aredetected, PPG Signal Generator 306 may generate a PPG signal/redchannel, as indicated by arrow 322, for further analysis and evaluationby Biomarker Detection module 310.

In some embodiments, the services and functionality of system 300 may becombined with different ground truth data to generate measures of otherhealth-related quantities and/or to provide greater insight into auser's health. For example, system 300 may be used to provide one ormore of the following measurements if the appropriate ground truth isavailable: Sp02, involving use of a hypoxia tent to lower the subject'sblood oxygen level and use of the Sp02 readings from a pulse oximeter;Blood glucose, involving the taking of blood glucose measurements via afinger stick and capturing data from a user before and after a meal;Hemoglobin A1C (HbA1c), involving taking a HbA1c measurement via afinger stick and lab analysis or point-of-care HbA1c analyzer; Heartrate variability, using an ECG device; Hemoglobin, using a hemoglobinmeasurement device; Respiration rate, by attaching a respiration rateband onto the user; and Arrhythmia, using an ECG device. Issue Detector308, Environment Assessor 312, and Signal Quality Assessor 314 describedabove as applied to blood pressure and heart rate measurements mayinstead or also be used to make measurements of Sp02, blood glucose,heart rate variability, hemoglobin, respiration rate, and arrhythmia.Further, in some embodiments, the camera (image/video sensor) of themobile device may be combined with other sensors on the device tocollect additional data and signals that may be used to assist inevaluating a user's health or condition.

As discussed above, biomarkers, such as blood pressure, hbA1c,hypertension risk, and heart rate, for example, may be estimated basedon biosensor data, such as a PPG. Optical biosensor data may be acquiredfrom a smartphone-captured video of a person's finger using a systemsuch as 300. For example, measurements of real-time blood pressure andheart rate data, or other biomarker, may be acquired via fingertip videoobtained with a mobile device. In this context, for example, “real-time”refers to a situation where such biomarkers may be provided to a userwhile the user continues to be measured for the biomarkers, the lattermeasurements perhaps providing updated measurements. In animplementation, a user may use a smartphone, including a DMM, to measuretheir blood pressure and heart rate by placing their fingertip againstthe lens of the smartphone camera.

In some embodiments, a PPG signal may be acquired from a smartphone byaveraging all pixels from the red channel of video of a fingertip fromthe smartphone. This process may produce a 1D (one-dimensional)time-series signal that mimics a PPG signal. However, this signal maygenerally be noisier than a signal acquired from a specially designeddevice like a pulse oximeter and may generally have a lower samplingrate. In one implementation, a region-of-interest (e.g., the middle200×200 pixels) may be used. In another implementation, the frames ofthe smartphone video may be used to create the 1D time-series PPG signalusing a deep learning model. Herein, a deep learning model refers to aneural network model that utilizes one or more of the following types oflayers: convolutional, residual, recurrent, attentional, andtransformer. For models operating on the smartphone video frames, theremay be two different types of deep learning models used: a 2D model thatoperates independently on each individual frame, and a 3D model thatoperates on two or more frames at one time. In both cases, the modelproduces a time-series of embeddings that may be used to produce anoutput, which may be another embedding, a prediction of a pulse oximetersignal, a biomarker estimation, and so on.

In one type of deep learning model, a baseline may be produced by abaseline biomarker estimation system that involves passing the averagedred channel signal or time-series of video frames through a deeplearning model to predict the biomarker. FIG. 4 illustrates a process402 that uses this type of baseline model, wherein the averaged redchannel 404 of input video frames 406 is used to create a 1D time-seriesnoisy signal 408 that resembles a PPG signal. This signal may then bepassed through a deep learning model 410 that uses embedding 412 toproduce an output 414, which may be predicted biomarkers such assystolic and diastolic blood pressure.

FIG. 5 illustrates another process that uses a different type ofbaseline model, wherein a time-series of video frames are passeddirectly through a deep learning model to predict the biomarker. Process502 provides a time-series of video frames 504 directly to a deeplearning model 506, which may use embedding 508 to produce an output510, which may be predicted biomarkers such as systolic and diastolicblood pressure.

In some embodiments, a deep learning model may be trained to usesmartphone data to predict a paired pulse oximeter signal. The input tothe model may be the 1D time-series smartphone data, described above,from the averaging of the red channel or computed from the frames of thevideo. The predicted pulse oximeter signal may then be used as input tothe deep learning model that will predict the biomarker.

FIG. 6 is a flow diagram of a process 600 for addressing signal qualityof a PPG signal, according to some embodiments. For example, a computerprocessing system (e.g., a processor) may perform process 600 thatincludes data gathering process 102 described above. At 602, theprocessor may acquire a series of images of tissue 104 using camera 106of a mobile device (e.g., smartphone 108). For example, clientapplication 302 may use Biomarker Detector UI 304 to operate the cameraof a mobile device.

At 604, the processor may transform the series of images into pseudo PPGsignals, as described above. At 606, the processor may assess the pseudoPPG signals to determine the quality of the pseudo PPG signals. Thepseudo PPG signal may have a signal characteristic associated with aquality indicator of the pseudo PPG signal. In other words, a feature ofthe pseudo PPG signal, such as the detailed shape of a waveform orpulse, for example, may allow for quantifying or characterizing thequality of the pseudo PPG signal. The feature may also allow fordetermining the circumstances (e.g., ambient conditions) that existedduring acquisition of the patient physiological data. Such a feature maybe compared to, or used in conjunction with, metadata gathered frommeasurements of light sensor 112 and accelerometer 114 duringacquisition.

At 608, the processor may determine whether to provide the pseudo PPGsignals to a deep learning model or to discard the pseudo PPG signalsbased on the determined quality. The processor may compare the signalcharacteristic of the pseudo PPG signal to a characteristic or parameterof a signal characteristic of a deep learning-based model PPG signal.For example, after being provided to Biomarker detection module 310, thesignal characteristic of the pseudo PPG signal may be compared to acharacteristic or parameter of an analogous signal characteristic of adeep learning-based model PPG signal.

At 610, the processor may measure or determine the disease state or thebiomarker based on the pseudo PPG signals provided to the deep learningmodel, as described above.

FIG. 7 is a flow diagram of a process 700 for determining signal qualityof a PPG signal and determining a disease state or biomarker based onthe PPG signal, according to some embodiments. For example, a computerprocessing system (e.g., a processor) may perform process 700 thatincludes data gathering process 102 described above. At 702, theprocessor may acquire a series of images of tissue 104 using camera 106of a mobile device (e.g., smartphone 108). For example, clientapplication 302 may use Biomarker Detector UI 304 to operate the cameraof a mobile device. At 704, the processor may transform the series ofimages into pseudo PPG signals, as described above. At 706, theprocessor may assign a quality value to the pseudo PPG signals based onthe quality of the pseudo PPG signals. In some implementations, thequality value may be a “pass” or “no pass” designation. In otherimplementations, the quality value may be a number (or a rating based ona number, such as lettering or other characters). The quality of thepseudo PPG signals may be determined by techniques described above.

At 708, the processor may provide to a deep learning model the pseudoPPG signals having the quality value exceeding a threshold value. Incontrast, at 710, the processor may discard the pseudo PPG signalshaving the quality value less than the threshold value. In other words,steps 708 and 710 sort out the pseudo PPG signals according to theirquality. If the quality is good, the pseudo PPG signals are used, but iftheir quality is bad, they are discarded. At 712, the processor maymeasure or determine the disease state or biomarker based on the pseudoPPG signals provided to the deep learning model.

In some implementations, in addition to measuring or determining adisease state or biomarker, an output may include displaying trends ofthe determined disease state or the biomarker over time. In someimplementations, an application (e.g., 302) may modify a medicationdosage based on the trends of the determined disease state or thebiomarker. The application may then display to a user the modifiedmedication dosage. In some cases, the application may analyze thedetermined disease state or the biomarker in conjunction with data basedon sleep quality, stress, or anxiety levels of a patient. The display orother portion of a UI may further include an alert to a user that thedetermined disease state or the biomarker exceeds a preset thresholdvalue. The display may also include a recommendation to engage in anactivity or to undertake a change in behavior, and the real-time bloodpressure and heart rate data.

FIG. 8 illustrates elements or components of a computer device or systemconfigured to implement a method, process, function, or operation ofsome embodiments of a system (e.g., 300) and methods described herein.As noted, in some embodiments, the system and methods may be implementedin the form of an apparatus that includes a processing element and a setof executable instructions. The executable instructions may be part of asoftware application and arranged into a software architecture.

In general, an embodiment may be implemented using a set of softwareinstructions that are designed to be executed by a suitably programmedprocessing element (such as a GPU, CPU, microprocessor, processor,controller, computing device, etc.). In a complex application or systemsuch instructions are typically arranged into “modules” with each suchmodule typically performing a specific task, process, function, oroperation. The entire set of modules may be controlled or coordinated intheir operation by an operating system (OS) or other form oforganizational platform.

Each application module or sub-module may correspond to a particularfunction, method, process, or operation that is implemented by executionof the instructions contained in the module or sub-module. Suchfunction, method, process, or operation may include those used toimplement one or more aspects, techniques, components, capabilities,steps, or stages of the described system and methods. In someembodiments, a subset of the computer-executable instructions containedin one module may be implemented by a processor in a first apparatus anda second and different subset of the instructions may be implemented bya processor in a second and different apparatus. This may happen, forexample, where a process or function is implemented by steps that occurin both a client device and a remote server.

The application modules and/or sub-modules may include any suitablecomputer executable code or set of instructions (e.g., as would beexecuted by a suitably programmed processor, microprocessor, or CPU),such as computer-executable code corresponding to a programminglanguage. For example, programming language source code may be compiledinto computer-executable code. Alternatively, or in addition, theprogramming language may be an interpreted programming language such asa scripting language.

The modules may contain one or more sets of instructions for performinga method or function described with reference to the figures. Thesemodules may include those illustrated but may also include a greaternumber or fewer number than those illustrated. As mentioned, each modulemay contain a set of computer-executable instructions. The set ofinstructions may be executed by a programmed processor contained in aserver, client device, network element, system, platform, or othercomponent.

A module may contain computer-executable instructions that are executedby a processor contained in more than one of a server, client device,network element, system, platform or other component. Thus, in someembodiments, a plurality of electronic processors, with each being partof a separate device, server, platform, or system may be responsible forexecuting all or a portion of the instructions contained in a specificmodule. Thus, although FIG. 8 illustrates a set of modules which takentogether perform multiple functions or operations, these functions oroperations may be performed by different devices or system elements,with certain of the modules (or instructions contained in those modules)being associated with those devices or system elements.

As illustrated in FIG. 8 , system 800 may represent a server or otherform of computing or data processing system, platform, or device.Modules 802 each contain a set of computer executable instructions,where when the set of instructions is executed by a suitable electronicprocessor or processors (such as that indicated in the figure by“Physical Processor(s) 830”), system (or server, platform, or device)800 operates to perform a specific process, operation, function, ormethod. Modules 802 are stored in a memory 820, which typically includesan Operating System module 804 that contains instructions used (amongother functions) to access and control the execution of the instructionscontained in other modules. The modules 802 stored in memory 820 areaccessed for purposes of transferring data and executing instructions byuse of a “bus” or communications line 816, which also serves to permitprocessor(s) 830 to communicate with the modules for purposes ofaccessing and executing a set of instructions. Bus or communicationsline 816 also permits processor(s) 830 to interact with other elementsof system 800, such as input or output devices 822, communicationselements 824 for exchanging data and information with devices externalto system 800, and additional memory devices 826.

For example, Module 806 may contain computer-executable instructionswhich when executed by a programmed processor cause the processor or adevice containing the processor to perform the functions of the clientapplication to enable the collection and processing of the acquired dataeither on the device or (as an example) to navigate to a server, uploadthe video or other data to the server, receive the output(s) of theblood pressure model from the server, and present the output(s) to theuser (in the form of a measure, graph, chart, range etc.). Module 808may contain computer-executable instructions which when executed by aprogrammed processor cause the processor or a device containing theprocessor to generate a blood pressure detector user interface forpresentation to a user. The interface may provide the user with toolsand instructions for collecting the video data using their device (suchas a smartphone).

The functionality for data acquisition, data processing, issuedetection, and the determination of blood pressure or anotherhealth-related quantity may be provided to a user in one or moreformats. These include an SDK that developers can use to perform one ormore of the functions (such as data collection, initial data processing,issue detection, running the blood pressure model) in their ownapplications. In another embodiment, a downloaded client application isprovided that is capable of performing one or more of the functionsdirectly to an end user. In another embodiment, certain functions of theclient application may be replaced by services accessible through anaccount on a SaaS platform.

In any of these embodiments, the trained models may reside as serviceson a remote server or platform so that the models can be updated andcontinually trained and improved as new data becomes available. Thisarrangement may allow anonymized data collected from multiple users tobe made available to other developers and model builders for use indetermining other health-related quantities and for determiningcorrelations between medical conditions, health, and the quantities. Inany of these embodiments, the interface may be provided through the web,mobile, or desktop, as long as there is an attached video camera.

Module 810 may contain computer-executable instructions which whenexecuted by a programmed processor cause the processor or a devicecontaining the processor to acquire the video data and subject it toprocessing (image and/or signal processing) to extract the red channelof the video, and to treat this signal as a pseudo-PPG signal. Module812 may contain computer executable instructions which when executed bya programmed processor cause the processor or a device containing theprocessor to monitor for issues or concerns that may occur during thevideo/data collection and to assist in determining if the collectedvideo/data is useable for generating the pseudo-PPG signal. Module 814may contain computer-executable instructions which when executed by aprogrammed processor cause the processor or a device containing theprocessor to perform functions used to generate a blood pressure measurefrom the acquired video in cases where the Issue Detector has notprevented further processing of the video frames. As illustrated in thefigure, this may include consideration of environmental conditions andsignal quality prior to inputting the pseudo-PPG signal (e.g., the redchannel of the video) into a trained blood pressure model. Module 815may contain computer-executable instructions which when executed by aprogrammed processor cause the processor or a device containing theprocessor to execute additional control logic for managing the decisionprocesses involved in obtaining device or camera parameters, settingparameters of the video/data collection process, and controlling thevideo/data processing flow, configuring a response to the determinedblood pressure measures, etc.

In some embodiments, certain of the functionality and services providedby the system and methods described herein may be made available tomultiple users by accessing an account maintained by a server or serviceplatform. Such a server or service platform may be termed a form ofSoftware-as-a-Service (SaaS).

It should be understood that the embodiments described above can beimplemented in the form of control logic using computer software in amodular or integrated manner. Based on the disclosure and teachingsprovided herein, a person of ordinary skill in the art will know andappreciate other ways and/or methods to implement the embodiments usinghardware and a combination of hardware and software.

In some embodiments, certain of the methods, models or functionsdescribed herein may be embodied in the form of a trained neuralnetwork, where the network is implemented by the execution of a set ofcomputer-executable instructions or representation of a data structure.The instructions may be stored in (or on) a non-transitorycomputer-readable medium and executed by a programmed processor orprocessing element. The set of instructions may be conveyed to a userthrough a transfer of instructions or an application that executes a setof instructions (such as over a network, e.g., the Internet). The set ofinstructions or an application may be utilized by an end-user throughaccess to a SaaS platform or a service provided through such a platform.A trained neural network, trained machine learning model, or other formof decision or classification process may be used to implement one ormore of the methods, functions, processes, or operations describedherein. Note that a neural network or deep learning model may becharacterized in the form of a data structure in which are stored datarepresenting a set of layers containing nodes, and connections betweennodes in different layers are created (or formed) that operate on aninput to provide a decision or value as an output.

In general terms, a neural network may be viewed as a system ofinterconnected artificial “neurons” that exchange messages between eachother. The connections have numeric weights that are “tuned” during atraining process, so that a properly trained network will respondcorrectly when presented with an image or pattern to recognize (forexample). In this characterization, the network consists of multiplelayers of feature-detecting “neurons”; each layer has neurons thatrespond to different combinations of inputs from the previous layers.Training of a network is performed using a “labeled” dataset of inputsin a wide assortment of representative input patterns that areassociated with their intended output response. Training usesgeneral-purpose methods to iteratively determine the weights forintermediate and final feature neurons. In terms of a computationalmodel, each neuron calculates the dot product of inputs and weights,adds the bias, and applies a non-linear trigger or activation function(for example, using a sigmoid response function).

Machine learning (ML) is being used more and more to enable the analysisof data and assist in making decisions in multiple industries. In orderto benefit from using machine learning, a machine learning algorithm isapplied to a set of training data and labels to generate a “model” whichrepresents what the application of the algorithm has “learned” from thetraining data. Each element (or example, in the form of one or moreparameters, variables, characteristics or “features”) of the set oftraining data is associated with a label or annotation that defines howthe element should be classified by the trained model. A machinelearning model is a set of layers of connected neurons that operate tomake a decision (such as a classification) regarding a sample of inputdata. When trained (i.e., the weights connecting neurons have convergedand become stable or within an acceptable amount of variation), themodel will operate on a new element of input data to generate thecorrect label or classification as an output.

An embodiment, which is provided as a particular example and is notintended to exclude other suitable methods, includes a methodology fortraining a biomarker estimation deep learning model, which may betrained on a large bank of biosignals (e.g., collected from public andprivate datasets). This may include PPG, ECG, seismocardiogram (SCG),ballistocardiogram (BCG), Electroencephalogram (EEG), and more. Theoutput of the model may be a series of embeddings that are utilized topredict some downstream task.

The training may utilize both unsupervised and semi-supervised learning.For unsupervised, portions of the signal may be removed from the inputsequence, and the model may be instructed to predict the missing portionof embeddings. In semi-supervised, certain characteristics of the signalmay be calculated (such as heart rate from the PPG signal), for whichthe model will attempt to predict these portions. In another iterationof semi-supervised, certain characteristics about samples that aresimilar will be calculated (e.g., from the same person, have the sameheart rate, etc.) and the model will be tasked with creating anembedding that is similar for these samples with the samecharacteristics.

The embeddings created by the model may then be used for biomarkerestimation. A deep learning model may operate on the embeddings createdby the large pre-trained model to predict the biomarker, which may besystolic BP and diastolic BP, for example.

For the large pretrained model, one may quantize the different values ofthe input signal. For example, for a signal that can take values of 0 to1, one may create 1 million equally spaced states, where each value ofthe signal is rounded to the nearest state.

As discussed above, there may be multiple sources of time-synchronizedsensor data collected at one time. For example, sources may include thepulse oximeter, IMU, ECG, or the smartphone optical and motion signals.These signals may be combined as separate channels of the input signal(e.g., channel 0=PPG, channel 1=accelerometer X, channel 2=accelerometerY). The deep learning model would then operate on this multi-channelsignal.

The deep learning model may be used to predict not only the biomarker ofinterest, but also additional features about the signal. For example,heart rate may be calculated from the PPG signal. The deep learningmodel would be instructed to predict both heart rate and the biomarkerof interest. The prediction of these additional features can improve theperformance of the biomarker prediction.

One such example is the prediction of pulse transit time (PTT). Using anIMU signal (motion) and optical signal, the time interval from when theaortic opening is sensed by the IMU and a landmark point in the opticalsignal (e.g., peak, valley, beginning of rise to systolic peak) may bemeasured. This time interval is PTT, which may be predicted by a modelthat contains both IMU and optical data. One may train different machinelearning models for different demographics. For example, one may train amodel for individuals of ages 0-40, 40-60, and 60+. During inference,one would use the model that matches the individual's demographics.

The data collected from the ear and toe pulse oximeters may be used tocalculate the PTT. By using a landmark point in both signals (e.g.,peak, valley, location of rise to systolic peak, etc.), the timeinterval between the landmark point in the ear and toe signals iscalculated. Next, a deep learning model running on either just theoptical fingertip signal or optical fingertip and IMU data may be usedto predict that PTT value.

An embodiment, which is provided as a particular example and is notintended to exclude other suitable methods, includes a methodology forcollecting training data for a model or models. A data collectionprocess may include, for example, a data collection person or persons.They first guide a subject through a relaxation period, such as aboutfive minutes, which may allow for stabilizing the subject's bloodpressure. Then they may take blood pressure readings (discarding thefirst) and alternative blood pressure readings and fingertip videos on avariety of smartphones. Throughout this process, the subject may beconnected to a pulse oximeter device to capture ground truth PPG. Theyalso may take demographic information that may affect the reading,including age, height, weight, biological sex, hand size, and skin tone.To ensure accuracy of the BP reading, BP may be measured viaauscultation and a digital stethoscope may be used along with a webcam.The webcam video may be spliced with stethoscope audio. A datacollection application may record fingertip videos. This application mayrecord fingertip video with frame-by-frame metadata that includes thecurrent camera settings for each frame. To align the PPG reading and asmartphone fingertip reading, a breadboard that has a button hooked upto an LED (e.g., a red LED) and to the pulse oximeter may be used. Thebutton activates the LED and sends a signal to the PPG device to startrecording. The smartphone is held up to the LED and the smartphone andPPG signals may be aligned based on that light. The data collectionprocess may be mobile to enable capture of training data from a greaterdiversity of subjects by going to them on-site rather than them havingto come to a specific location. Once data is collected, it may beuploaded into cloud storage. Processing the data may begin once it'suploaded, automatically syncing the PPG and smartphone fingertip reading(by looking for a spike in the red), clipping the smartphone and PPGsignals to just the relevant portions with the fingertip on the camera(by looking for ratio of red to blue and green), fine-tuning thealignment of the PPG and smartphone signal, scoring the smartphonesignals, and producing plots that can be analyzed. This data can be usedto train the model(s). In addition to preparing data and producingplots, a database may also be produced of individual pulses that includesignal quality score info and demographic data that we can use to debuga model quickly. The system may potentially use the demographic data asinput into the BP model for greater accuracy as well, and for segmentingusers to provide more accurate BP values.

In some embodiments, the system or services described herein may beimplemented as microservices, processes, workflows or functionsperformed in response to the submission of a set of input data. Themicroservices, processes, workflows or functions may be performed by aserver, data processing element, platform, or system. In someembodiments, the data analysis and other services may be provided by aservice platform located “in the cloud”. In such embodiments, theplatform may be accessible through APIs and SDKs. The functions,processes and capabilities described herein and with reference to one ormore of the figures may be provided as microservices within theplatform. The interfaces to the microservices may be defined by REST andGraphQL endpoints. An administrative console may allow users or anadministrator to securely access the underlying request and responsedata, manage accounts and access, and in some cases, modify theprocessing workflow or configuration.

Note that although some embodiments described herein are directed to amulti-tenant or SaaS architecture that may be used for the delivery ofbusiness-related or other applications and services to multipleaccounts/users, such an architecture may also be used to deliver othertypes of data processing services and provide access to otherapplications. For example, such an architecture may be used to provideone or more of the processes, functions, and operations describedherein. Although in some embodiments, a platform or system of the typeillustrated in the figures may be operated by a 3rd party provider toprovide a specific set of services or applications, in otherembodiments, the platform may be operated by a provider and a differententity may provide the applications or services for users through theplatform.

FIG. 9 illustrates an SaaS system 900 in which an embodiment may beimplemented or through which an embodiment of the services describedherein may be accessed. In accordance with the advantages of anapplication service provider (ASP) hosted business service system (suchas a multi-tenant data processing platform), users of the servicesdescribed herein may comprise individuals, businesses, stores,organizations, etc. A user may access the services using any suitableclient, including but not limited to desktop computers, laptopcomputers, tablet computers, scanners, smartphones, etc. In general, anyclient device having access to the Internet may be used to provide datato the platform for processing and evaluation. A user interfaces withthe service platform across the Internet 908 or another suitablecommunications network or combination of networks. Examples of suitableclient devices include desktop computers 903, smartphones 904, tabletcomputers 905, or laptop computers 906.

System 900, which may be hosted by a third party, may include a set ofdata analysis and other services to assist in acquiring and processingvideo data, and generating a measure or measures of one or morehealth-related quantities, and a web interface server 914. The servicesmay include one or more functions or operations for the processing ofacquired video or other data to enable the generation of a measure ofblood pressure or other health-related quantity. As examples, in someembodiments, the set of functions, operations or services made availablethrough the platform or system 900 may include Account Managementservices 916, which may implement a process or service to authenticate auser wishing to submit an example of video data and from that determinea measure of a health-related quantity, or which may implement a processor service to generate a container or instantiation of the dataprocessing and analysis services for that user. System 900 may alsoinclude Initial Data Acquisition and Processing services 917, which mayimplement a process or service to provide a client-side application to auser for installation on their device, may implement a process orservice to receive a set of frames of video data from the user device;or may implement a process or service to process each frame to extract ared or pseudo-PPG channel (the PPG Signal Generator)

System 900 may also include Identify/Detect Video Collection Issues ofPossible Concern services 918, which may implement a process or serviceto monitor and detect one or more situations or events that mightindicate the video data acquired during that situation or event is notreliable or accurate, or may implement a process or service to executelogic to control the collection or processing of the video frames basedon the presence or absence of such a situation or event. System 900 mayalso include Determine Blood Pressure Measure(s) and/or OtherHealth-Related Quantities services 919, which may implement a process orservice to assess environmental factors and signal quality factors (inone embodiment using a trained signal quality assessment model), and ifthose are acceptable, to provide the generated (extracted) PPG signal toa trained model. Determine Blood Pressure Measure(s) and/or OtherHealth-Related Quantities services 919 may also implement a trainedmodel that operates to receive the generated PPG signal extracted fromthe video frames and in response to generate a measure or measures ofone or more health related quantities, such as blood pressure.

System 900 may also include Control Logic services 920, which mayimplement a process or service to implement other needed control logicto control the acquisition, processing, and use of the trained model(s),may implement a process or service to implement other logic fordelivering and presenting the output of the trained blood pressure modelto a user, including trend data and/or other forms of analysis of bloodpressure measures, or may implement a process or service to implementlogic to process data from other sensors and/or to process such data togenerate other forms of signals for input to a trained model. System 900may also include administrative services 922, which may implement aprocess or services to provide platform and services administration,such as, for example, enabling the provider of the services and/or theplatform to administer and configure the processes and services providedto users.

The platform or system illustrated in FIG. 9 may be hosted on adistributed computing system made up of at least one, but likelymultiple, “servers.” A server is a physical computer dedicated toproviding data storage and an execution environment for one or moresoftware applications or services intended to serve the needs of theusers of other computers that are in data communication with the server,for instance via a public network such as the Internet. The server, andthe services it provides, may be referred to as the “host” and theremote computers, and the software applications running on the remotecomputers being served may be referred to as “clients.” Depending on thecomputing service(s) that a server offers it could be referred to as adatabase server, data storage server, file server, mail server, printserver, web server, etc. A web server is most often a combination ofhardware and the software that helps deliver content, commonly byhosting a website, to client web browsers that access the web server viathe Internet.

FIG. 10 illustrates elements or components of an example operatingenvironment 1000 in which embodiments described herein may beimplemented. As illustrated, a variety of clients 1002 incorporatingand/or incorporated into a variety of computing devices may communicatewith a multi-tenant service platform 1008 through one or more networks1014. For example, a client may incorporate and/or be incorporated intoa client application (e.g., software) implemented at least in part byone or more of the computing devices. Examples of suitable computingdevices include personal computers, server computers 1004, desktopcomputers 1006, laptop computers 1007, notebook computers, tabletcomputers or personal digital assistants (PDAs) 1010, smart phones 1012,cell phones, and consumer electronic devices incorporating one or morecomputing device components, such as one or more electronic processors,microprocessors, central processing units (CPU), or controllers.Examples of suitable networks 1014 include networks utilizing wiredand/or wireless communication technologies and networks operating inaccordance with any suitable networking and/or communication protocol(e.g., the Internet).

The distributed computing service/platform (which may also be referredto as a multi-tenant data processing platform) 1008 may include multipleprocessing tiers, including a user interface tier 1016, an applicationserver tier 1020, and a data storage tier 1024. The user interface tier1016 may maintain multiple user interfaces 1017, including graphicaluser interfaces and/or web-based interfaces. The user interfaces mayinclude a default user interface for the service to provide access toapplications and data for a user or “tenant” of the service (depicted as“Service UI” in the figure), as well as one or more user interfaces thathave been specialized/customized in accordance with user specificrequirements (e.g., represented by “Tenant A UI”, . . . , “Tenant Z UI”in the figure, and which may be accessed via one or more APIs).

The default user interface may include user interface componentsenabling a tenant to administer the tenant's access to and use of thefunctions and capabilities provided by the service platform. This mayinclude accessing tenant data, launching an instantiation of a specificapplication, causing the execution of specific data processingoperations, etc. Each application server or processing tier 1022illustrated in the figure may be implemented with a set of computersand/or components including computer servers and processors, and mayperform various functions, methods, processes, or operations asdetermined by the execution of a software application or set ofinstructions. The data storage tier 1024 may include one or more datastores, which may include a Service Data store 1025 and one or moreTenant Data stores 1026. Data stores may be implemented with anysuitable data storage technology, including structured query language(SQL) based relational database management systems (RDBMS).

Service Platform 1008 may be multi-tenant and may be operated by anentity to provide multiple tenants with a set of business-related orother data processing applications, data storage, and functionality. Forexample, the applications and functionality may include providingweb-based access to the functionality used by a business to provideservices to end-users, thereby allowing a user with a browser and anInternet or intranet connection to view, enter, process, or modifycertain types of information. Such functions or applications aretypically implemented by one or more modules of softwarecode/instructions that are maintained on and executed by one or moreservers 1022 that are part of the platform's Application Server Tier1020. As noted with regards to FIG. 9 , the platform system illustratedin FIG. 10 may be hosted on a distributed computing system made up of atleast one, but typically multiple, “servers.”

As mentioned, rather than build and maintain such a platform or systemthemselves, a business may utilize systems provided by a third party. Athird party may implement a business system/platform as described abovein the context of a multi-tenant platform, where individualinstantiations of a business' data processing workflow (such as the dataanalysis and evaluation services and processing described herein) areprovided to users, with each business representing a tenant of theplatform. One advantage to such multi-tenant platforms is the abilityfor each tenant to customize their instantiation of the data processingworkflow to that tenant's specific business needs or operationalmethods. Each tenant may be a business or entity that uses themulti-tenant platform to provide business services and functionality tomultiple users.

FIG. 11 illustrates additional details of the elements or components ofthe multitenant distributed computing service platform of FIG. 10 , inwhich an embodiment may be implemented. The software architectureillustrated in FIG. 11 represents an example of an architecture whichmay be used to implement an embodiment. In general, an embodiment may beimplemented using a set of software instructions that are designed to beexecuted by a suitably programmed processing element (such as a CPU,microprocessor, processor, controller, computing device, etc.). In acomplex system such instructions are typically arranged into “modules”with each such module performing a specific task, process, function, oroperation. The entire set of modules may be controlled or coordinated intheir operation by an operating system (OS) or other form oforganizational platform.

As noted, FIG. 11 is a diagram illustrating additional details of theelements or components 1100 of a multi-tenant distributed computingservice platform, in which an embodiment may be implemented. The examplearchitecture includes a user interface layer or tier 1102 having one ormore user interfaces 1103. Examples of such user interfaces includegraphical user interfaces and APIs. Each user interface may include oneor more interface elements 1104. For example, users may interact withinterface elements to access functionality and/or data provided byapplication and/or data storage layers of the example architecture.Examples of graphical user interface elements include buttons, menus,checkboxes, drop-down lists, scrollbars, sliders, spinners, text boxes,icons, labels, progress bars, status bars, toolbars, windows,hyperlinks, and dialog boxes. Application programming interfaces may belocal or remote and may include interface elements such as parameterizedprocedure calls, programmatic objects, and messaging protocols.

The application layer 1110 may include one or more application modules1111, each having one or more sub-modules 1112. Each application module1111 or sub-module 1112 may correspond to a function, method, process,or operation that is implemented by the module or sub-module (e.g., afunction or process related to providing business related dataprocessing and services to a user of the platform). Such function,method, process, or operation may include those used to implement one ormore aspects of the inventive system and methods, such as for one ormore of the processes or functions described with reference to thefigures.

The application modules and/or sub-modules may include any suitablecomputer-executable code or set of instructions (e.g., as would beexecuted by a suitably programmed processor, microprocessor, or CPU),such as computer-executable code corresponding to a programminglanguage. For example, programming language source code may be compiledinto computer-executable code. Alternatively, or in addition, theprogramming language may be an interpreted programming language such asa scripting language. Each application server (e.g., as represented byelement 1022 of FIG. 10 ) may include each application module.Alternatively, different application servers may include different setsof application modules. Such sets may be disjoint or overlapping.

The data storage layer 1120 may include one or more data objects 1122each having one or more data object components 1121, such as attributesand/or behaviors. For example, the data objects may correspond to tablesof a relational database, and the data object components may correspondto columns or fields of such tables. Alternatively, or in addition, thedata objects may correspond to data records having fields and associatedservices. Alternatively, or in addition, the data objects may correspondto persistent instances of programmatic data objects, such as structuresand classes. Each data store in the data storage layer may include eachdata object. Alternatively, different data stores may include differentsets of data objects. Such sets may be disjoint or overlapping.

Note that the example computing environments depicted in FIGS. 8-11 arenot intended to be limiting examples. Further environments in which anembodiment may be implemented in whole or in part include devices(including mobile devices), software applications, systems, apparatuses,networks, SaaS platforms, IaaS (infrastructure-as-a-service) platforms,or other configurable components that may be used by multiple users fordata entry, data processing, application execution, or data review.

Any of the software components, processes or functions described in thisapplication may be implemented as software code to be executed by aprocessor using any suitable computer language such as Python, Java,JavaScript, C++ or Perl using conventional or object orientedtechniques. The software code may be stored as a series of instructions,or commands in (or on) a non-transitory computer-readable medium, suchas a random-access memory (RAM), a read only memory (ROM), a magneticmedium such as a hard-drive or a floppy disk, or an optical medium suchas a CD-ROM. In this context, a non-transitory computer-readable mediumis almost any medium suitable for the storage of data or an instructionset aside from a transitory waveform. Any such computer readable mediummay reside on or within a single computational apparatus and may bepresent on or within different computational apparatuses within a systemor network.

According to one example implementation, the term processing element orprocessor, as used herein, may be a central processing unit (CPU), orconceptualized as a CPU (such as a virtual machine). In this exampleimplementation, the CPU or a device in which the CPU is incorporated maybe coupled, connected, and/or in communication with one or moreperipheral devices, such as display. In another example implementation,the processing element or processor may be incorporated into a mobilecomputing device, such as a smartphone or tablet computer.

The non-transitory computer-readable storage medium referred to hereinmay include a number of physical drive units, such as a redundant arrayof independent disks (RAID), a floppy disk drive, a flash memory, a USBflash drive, an external hard disk drive, thumb drive, pen drive, keydrive, a High-Density Digital Versatile Disc (HD-DVD) optical discdrive, an internal hard disk drive, a Blu-Ray optical disc drive, or aHolographic Digital Data Storage (HDDS) optical disc drive, synchronousdynamic random access memory (SDRAM), or similar devices or other formsof memories based on similar technologies. Such computer-readablestorage media allow the processing element or processor to accesscomputer-executable process steps, application programs and the like,stored on removable and non-removable memory media, to off-load datafrom a device or to upload data to a device. As mentioned, with regardsto the embodiments described herein, a non-transitory computer-readablemedium may include almost any structure, technology, or method apartfrom a transitory waveform or similar medium.

Certain implementations of the disclosed technology are described hereinwith reference to block diagrams of systems, and/or to flowcharts orflow diagrams of functions, operations, processes, or methods. It willbe understood that one or more blocks of the block diagrams, or one ormore stages or steps of the flowcharts or flow diagrams, andcombinations of blocks in the block diagrams and stages or steps of theflowcharts or flow diagrams, respectively, may be implemented bycomputer-executable program instructions. Note that in some embodiments,one or more of the blocks, or stages or steps may not necessarily needto be performed in the order presented or may not necessarily need to beperformed at all.

These computer-executable program instructions may be loaded onto ageneral purpose computer, a special purpose computer, a processor, orother programmable data processing apparatus to produce a specificexample of a machine, such that the instructions that are executed bythe computer, processor, or other programmable data processing apparatuscreate means for implementing one or more of the functions, operations,processes, or methods described herein. These computer programinstructions may also be stored in a computer readable memory that maydirect a computer or other programmable data processing apparatus tofunction in a specific manner, such that the instructions stored in thecomputer-readable memory produce an article of manufacture includinginstruction means that implement one or more of the functions,operations, processes, or methods described herein.

Other objects and advantages of the systems and methods described willbe apparent to one of ordinary skill in the art upon review of thedetailed description and the included figures. Throughout the drawings,identical reference characters and descriptions indicate similar, butnot necessarily identical, elements. While the exemplary embodimentsdescribed herein are susceptible to various modifications andalternative forms, specific embodiments have been illustrated by way ofexample in the drawings and will be described in detail herein. However,the exemplary embodiments described herein are not intended to belimited to the particular forms disclosed. Rather, the presentdisclosure covers all modifications, equivalents, and alternativesfalling within the scope of the appended claims.

The use of the terms “a” and “an” and “the” and similar referents in thespecification and in the following claims are to be construed to coverboth the singular and the plural, unless otherwise indicated herein orclearly contradicted by context. The terms “having,” “including,”“containing” and similar referents in the specification and in thefollowing claims are to be construed as open-ended terms (e.g., meaning“including, but not limited to,”) unless otherwise noted. Recitation ofranges of values herein are merely indented to serve as a shorthandmethod of referring individually to each separate value inclusivelyfalling within the range, unless otherwise indicated herein, and eachseparate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein may beperformed in any suitable order unless otherwise indicated herein orclearly contradicted by context. The use of any and all examples, orexemplary language (e.g., “such as”) provided herein, is intended merelyto better illuminate embodiments and does not pose a limitation to thescope of the embodiments unless otherwise claimed. No language in thespecification should be construed as indicating any non-claimed elementas essential to each embodiment.

As used herein (e.g., the claims, figures, and specification), the term“or” is used inclusively to refer items in the alternative and incombination.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the disclosure.However, it will be apparent to one skilled in the art that the specificdetails are not required in order to practice the systems and methodsdescribed herein. The foregoing descriptions of specific embodiments orexamples are presented by way of examples for purposes of illustrationand description. They are not intended to be exhaustive of or to limitthis disclosure to the precise forms described. Many modifications andvariations are possible in view of the above teachings. The embodimentsor examples are illustrated and described in order to best explain theprinciples of this disclosure and practical applications, to therebyenable others skilled in the art to best utilize this disclosure andvarious embodiments or examples with various modifications as are suitedto the particular use contemplated. It is intended that the scope ofthis disclosure be defined by the following claims and theirequivalents.

1. A method of measuring or determining a disease state or a biomarker,the method comprising: acquiring a series of images of a tissue using acamera of a mobile device; transforming the series of images into pseudophotoplethysmography (PPG) signals; assessing the pseudo PPG signals todetermine the quality of the pseudo PPG signals; determining whether toprovide the pseudo PPG signals to a deep learning model or to discardthe pseudo PPG signals based on the determined quality; and measuring ordetermining the disease state or the biomarker based on the pseudo PPGsignals provided to the deep learning model.
 2. The method of claim 1,wherein the series of images comprises a video signal.
 3. The method ofclaim 1, wherein acquiring the series of images includes controlling aflash and exposure settings of the camera.
 4. The method of claim 1,wherein discarding the pseudo PPG signals comprises: not providing thepseudo PPG signals to the deep learning model; adjusting the flash orthe exposure settings of the camera; and initiating additional acquiringof images of the tissue using the adjusted flash or exposure settings.5. The method of claim 1, further comprising: measuring ambient lightreaching the tissue using a photosensor onboard the mobile device;measuring relative motion between the tissue and the camera using anaccelerometer onboard the mobile device; and grouping the measuredambient light and the measured relative motion as metadata.
 6. Themethod of claim 5, wherein the quality of the pseudo PPG signals isbased, at least in part, on the metadata.
 7. The method of claim 1,wherein assessing the pseudo PPG signals to determine the quality of thepseudo PPG signals further comprises analyzing shapes and features ofwaveforms of the pseudo PPG signals, wherein the quality of the pseudoPPG signals is based, at least in part, on the shapes and features ofthe waveforms.
 8. The method of claim 1, wherein the biomarker comprisesmeasures of blood pressure, heart rate, SpO2, or other health-relatedquantity.
 9. The method of claim 1, wherein the mobile device is asmartphone, wrist-worn wearable such as smartwatch or smart wristband,earbud, or tablet.
 10. A method of measuring or determining a diseasestate or biomarker, the method comprising: acquiring a series of imagesof a tissue using a camera of a mobile device; transforming the seriesof images into pseudo photoplethysmography (PPG) signals; assigning aquality value to the pseudo PPG signals based on the quality of thepseudo PPG signals; providing to a deep learning model the pseudo PPGsignals having the quality value exceeding a threshold value; discardingthe pseudo PPG signals having the quality value less than the thresholdvalue; and measuring or determining the disease state or biomarker basedon the pseudo PPG signals provided to the deep learning model.
 11. Themethod of claim 10, wherein transforming the series of images into thePPG signals comprises applying a second deep learning model to theseries of images, wherein an output of the second deep learning model isthe PPG signals.
 12. The method of claim 10, wherein the series ofimages comprises a video signal.
 13. The method of claim 10, whereinacquiring the series of images includes controlling a flash and exposuresettings of the camera.
 14. The method of claim 13, wherein discardingthe pseudo PPG signals comprises: not providing the pseudo PPG signalsto the deep learning model; adjusting the flash or the exposure settingsof the camera; and initiating additional acquiring of images of thetissue using the adjusted flash or exposure settings.
 15. The method ofclaim 10, further comprising: measuring ambient light reaching thetissue using a photosensor onboard the mobile device; measuring relativemotion between the tissue and the camera using an accelerometer onboardthe mobile device; and grouping the measured ambient light and themeasured relative motion as metadata.
 16. The method of claim 15,wherein the quality of the pseudo PPG signals is based, at least inpart, on the metadata.
 17. The method of claim 10, wherein assessing thepseudo PPG signals to determine the quality of the pseudo PPG signalsfurther comprises analyzing shapes and features of waveforms of thepseudo PPG signals, wherein the quality of the pseudo PPG signals isbased, at least in part, on the shapes and features of the waveforms.18. The method of claim 10, wherein the biomarker comprises measures ofblood pressure and heart rate.
 19. The method of claim 10, wherein thebiomarker comprises measures SpO2 or other health-related quantity. 20.The method of claim 10, wherein the mobile device is a smartphone,wrist-worn wearable such as smartwatch or smart wristband, earbud, ortablet.